Inferring spatial relations from textual descriptions of images

Generating an image from its textual description requires both a certain level of language understanding and common sense knowledge about the spatial relations physical entities being described. In this work, we focus on inferring relation between entities, key step in process composing scenes based text. More specifically, given caption containing mention to subject location size bounding box that subject, our goal is predict object mentioned caption. Previous work did not use text information, but manually provided holding object. fact, used evaluation datasets contain annotated ontological triplets no captions, making exercise unrealistic: manual was required; systems leverage richer information captions. Here present system uses full caption, Relations Captions (REC-COCO), dataset derived MS-COCO which allows evaluate inference captions directly. Our experiments show that: (1) it possible infer with respect directly caption; (2) place better than using relation. paves way for that, decide need be depicted their respective sizes, order then generate final image.